Summary

Stale node_models rows (state="loaded") were surviving past the healthy state of their owning node, causing /embeddings (and other inference paths) to dispatch to a backend whose process was gone or drained. The downstream symptom was pgvector rejecting inserts as vector cannot have more than 16000 dimensions (SQLSTATE 54000) because the misbehaving backend silently returned a malformed (oversized) tensor; the Models page showed the model as "running" without an associated node, like a stale entry, even though the node was no longer visible in the Nodes view.

Why

Repro path observed live:

• qwen3-embedding-4b was registered on a worker that had been operationally drained (process gone, but node-level heartbeats still arriving).
• FindAndLockNodeWithModel picked the stale node_models row because the JOIN only filtered by node_models.state = "loaded"; the status-check happened in a second query whose failure rolled back the transaction but didn't prevent the same row from being picked next time.
• probeHealth (gRPC ping) passed against the drained worker.
• The backend returned an oversized tensor → /embeddings returned 200 → LocalRecall's pgvector insert blew up at the 16000-dim ceiling.
• MarkUnhealthy / MarkDraining only flipped backend_nodes.status; unlike MarkOffline, they didn't cascade-clean node_models. Stale rows accumulated.

Changes (defense-in-depth)

• MarkUnhealthy / MarkDraining cascade-delete node_models — mirrors MarkOffline. On the unhealthy path the backend is presumed unreachable; on drain, routing already filters these out, but clearing the rows stops the UI from misreporting and prevents future scheduling-logic relaxations from picking them up.
• FindAndLockNodeWithModel inner JOIN now also filters by backend_nodes.status = healthy — the previous version relied on the second node-fetch step to reject non-healthy nodes, but a concurrent reader could still pick the same stale row in the same window. Belt-and-braces.
• DistributedConfig.PerModelHealthCheck → DisablePerModelHealthCheck (inverted; on by default) — the per-model gRPC probe independently health-checks each model's gRPC address and removes stale node_models rows when the backend has crashed even though the worker's node-level heartbeat is still arriving.

Migration

DistributedConfig.PerModelHealthCheck was renamed and inverted to DisablePerModelHealthCheck. The field had no CLI flag, env var binding, or YAML key in tree (only the bare struct field), so there's no user-facing migration. Anything constructing DistributedConfig in code needs to drop the assignment (default now does the right thing) or invert it.

Test plan

• go test ./core/services/nodes/ (100s, all pass)
• go vet ./core/services/nodes/ ./core/config/ ./core/application/ clean
• Live verification in the dev cluster where the original symptom was observed: drain a backend hosting an embedding model, confirm /embeddings either returns an error (no replica) or routes to a healthy one — no 200-with-garbage, no pgvector dimension overflow downstream.

🤖 Generated with Claude Code